Skip to content

feat: held-out validation during training#77

Open
ZouzouWP wants to merge 1 commit into
huggingface:mainfrom
ZouzouWP:feat/eval-validation
Open

feat: held-out validation during training#77
ZouzouWP wants to merge 1 commit into
huggingface:mainfrom
ZouzouWP:feat/eval-validation

Conversation

@ZouzouWP

Copy link
Copy Markdown

Until now, training loss was the only signal available during a run — no way to tell a policy was starting to overfit without stopping and evaluating it separately afterward, by which point the ideal checkpoint was already steps behind.

What changed

  • New "Validation" section in training's advanced config: a single toggle drives dataset.eval_split (fraction of episodes held out from training) and eval_steps (evaluation cadence) together, since lerobot requires both or neither.
  • Backend parses step N: eval_loss=X lines (emitted by lerobot_train whenever eval_steps > 0) into a new eval_loss field, exposed on both the live training status and the persisted metrics history (so the curve survives a page reload).
  • The monitoring chart plots eval_loss as a second, sparser line alongside the training loss, so overfitting becomes visible while training is still running instead of only in hindsight.
flowchart TB
    A["Held-out episodes\n(e.g. 10%)"] --> B["Periodic evaluation"]
    C["Training loop"] --> D["Training loss\n(every step)"]
    B --> E["Validation loss\n(every N steps)"]
    D --> F["Monitoring chart"]
    E --> F
Loading

Testing

Implemented and code-reviewed against the log-parsing regex and the chart's dedupe/merge logic, but not yet run end-to-end with eval_steps > 0 against a real training job — needs a validation pass to confirm the parsed values match lerobot_train's actual output format before merge.

Until now, training loss was the only signal available during a run — no way to tell a policy was starting to overfit without stopping and evaluating it separately afterward, by which point the ideal checkpoint was already steps behind.

## What changed

- New "Validation" section in training's advanced config: a single toggle drives `dataset.eval_split` (fraction of episodes held out from training) and `eval_steps` (evaluation cadence) together, since `lerobot` requires both or neither.
- Backend parses `step N: eval_loss=X` lines (emitted by `lerobot_train` whenever `eval_steps > 0`) into a new `eval_loss` field, exposed on both the live training status and the persisted metrics history (so the curve survives a page reload).
- The monitoring chart plots `eval_loss` as a second, sparser line alongside the training loss, so overfitting becomes visible while training is still running instead of only in hindsight.

```mermaid
flowchart TB
    A["Held-out episodes\n(e.g. 10%)"] --> B["Periodic evaluation"]
    C["Training loop"] --> D["Training loss\n(every step)"]
    B --> E["Validation loss\n(every N steps)"]
    D --> F["Monitoring chart"]
    E --> F
```

## Testing

Implemented and code-reviewed against the log-parsing regex and the chart's dedupe/merge logic, but **not yet run end-to-end** with `eval_steps > 0` against a real training job — needs a validation pass to confirm the parsed values match `lerobot_train`'s actual output format before merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant